Repository navigation
Conversation
Math.max(...lengths) throws RangeError past ~100k args, so any large batch (bulk triage, big shortlist embed) crashed instead of running. Replace every spread-max on batch-controlled lengths with explicit loops, and while there remove the flat(Infinity)+map intermediates on the encoder->head path, which allocated the [B,L,H] hidden tensor up to three times per forward pass. - common.ts: new maxOf(values, fallback) loop helper; collateItems derives L/K with loops instead of Math.max(...maps) - providers.ts: feed/feedHead derive L/k with loops; toNested copies once with precomputed steps (no per-node slice/reduce); i64, hidden_states and marker_mask flatten straight into their typed arrays instead of flat(Infinity).map + TypedArray.from - shortlist.ts: embed batch width via maxOf No behavior change: tensor values and dims are identical, verified by tests/flat-parity.test.ts. New tests/maxlen.test.ts proves a 300k-row length list no longer throws. Tests: 167 passed (incl. 4 new), tsc build clean.
Three hot paths redo identical work on every call: BPE merges run O(L^2) per word even for words seen a thousand times, question prefixes re-encode the same instructions per state, and every predict allocates hook arrays that are empty in the common no-hooks case. All three are transparent to callers. - tokenizer.ts: per-merges word->pieces cache in bpeWord (cap 20k, cleared on overflow); single-char inputs keep keys collision-free - common.ts: per-tokenizer question-prefix cache keyed by the full question JSON (cap 1k); sequenceWithState never mutates prefixes, so sharing the cached object is safe - hooks.ts: composeHooks returns a shared empty list when no default, installed, or per-call hooks exist; dispatch only reads it Correctness is guarded by tests/prefix-cache.test.ts (repeat call returns the cached prefix) plus the existing BPE, sequence-prefix, and hooks suites, which all pass unchanged. Tests: 170 passed (incl. 1 new), tsc build clean.
Bulk triage ran N sequential forward passes for N states. Now one encoder+head run covers all S*Q rows: prefixes build once per question, per-state tokens are summed for usage, and answers decode per (state, question) row. Hook semantics are unchanged -- start hooks still see and may rewrite all states, skip() still short-circuits, and end hooks still see aggregated usage. - agent.ts: predictMany/systemMany; _predictHooked batches via the new _systemManyCore; _systemOneCore delegates to it, so single and batch paths share one implementation and stay identical - router.ts: predictMany/systemMany route each state (firing onRoute per state), group indices by model, load each checkpoint once, keep input order, and attach per-state routing; agents without predictMany fall back to looped systemOne; a start hook that changes the state count triggers re-routing so decisions stay aligned - tests/batch.test.ts: single encoder run for N states, batch equals looped systemOne answers, empty inputs, router grouping with one load per model and order preservation Verified live on the shipped multilingual weights (try-batch.mjs): 3 states in one pass, routing english/multilingual/multilingual. Tests: 175 passed (incl. 5 new), tsc build clean.
Two integration papercuts: building a valid question dict means remembering the exact type/criteria shapes, and predictShortlist required callers to hand-roll an embed_fn even though the agent already carries an encoder. Both now have one-line answers. - dx.ts (new, exported from the package root alongside maxOf): q.choice/q.score/q.noul factories whose output passes checkQuestion; best() argmax over choice/score/noul answers; topK() probability ranking; isConfident() gate (default 0.8) - shortlist.ts: predictShortlist embedFn becomes optional; when a choice question actually needs shortlisting (k < n), it defaults to embedFnFromAgent(agent), which throws honestly if the agent has no encoder access. Passthrough questions never touch an encoder. - tests/dx.test.ts: builder validity, helper semantics, and auto-embed end to end with a fake agent Backwards compatible: explicit embedFn callers are unaffected. Tests: 179 passed (incl. 4 new), tsc build clean.
Model downloads failed hard on the first transient error, could not be cancelled, and gave no feedback while multi-hundred-MB weights transferred. Loads now take signal/onProgress end to end, and the browser cache key is versioned. - providers.ts: fetchWithRetry (2 retries on network errors and 429/5xx, 404s fail fast, aborts never retried); signal and onProgress(done, total, file) plumbed through loadNodeBundle, loadWebBundle, and Agent.load; CacheStorage key laya-ts-v1; shared TOKENIZER_CANDIDATES for the node/web tokenizer layouts (full loader unification skipped: 8 lines of duplication cost less than an IO-injection abstraction) - agent.ts: Agent.load accepts and forwards signal/onProgress - tests/load-robustness.test.ts: 503-then-success retry plus progress callbacks; pre-aborted signal throws without retrying Skipped deliberately: cache TTL (online-first plus the browser HTTP cache already covers freshness). Tests: 181 passed (incl. 2 new), tsc build clean.
try-ml.mjs and try-batch.mjs hardcoded ./model-ml, which exists in no checkout: running either failed with "Incompatible model" and fell into a Hugging Face download of a bogus path. snake.mjs had the same default with no fallback. - examples/try-batch.mjs (new): predictMany over English/Hindi/German states in one forward pass, then Router.predictMany with per-state routing; prints choice + confidence + full distribution per state; attaches the local weights for both languages so the demo runs offline - examples/try-ml.mjs, examples/snake.mjs: --model flag plus first- existing-dir fallback (./model-ml, laya-ts/examples/model-ml, ./examples/model-ml) so every example finds the shipped weights from either the repo root or the laya-ts dir - README.md: try-batch run line and a 5-line predictMany/q-helper snippet Verified live on the shipped multilingual weights: batched billing/billing/other with routing english/multilingual/multilingual; snake --ticks 5 runs on model weights; try-ml reports billing 0.99 on the Hindi refund request.
|
@NandhaKishorM take a look |
|
Thank you for all the work here, several of these pieces are genuinely valuable. We are going with #329 for batch inference because it mirrors Python's predict_batch names and per-request hook behaviour, so please drop predictMany and systemMany. Could you split the rest into small focused PRs: first the maxOf spread-overflow fix with the typed-array flattening and its parity test, then the retrying load with abort and progress, then the example path fixes. For the helpers, please have isConfident read answer_confidence (now in main via #386) rather than the entropy confidence, and include a quick measurement with the caches so we can judge them on their own. |
|
Still holding on the split, which has not happened, and I want to restate what I would take so this is not just a stale request. The decision on batch inference stands: #329 shipped it, because it mirrors Python's The rest I do want, in this order, as separate PRs:
Note that The branch conflicts now and splitting it is less work than resolving it. Thanks for the volume of work in here; the sequencing is the only thing in the way. |
|
Closing this one. It is 2,187 commits behind Specifically, on batch inference: the TypeScript SDK has I am not going to pretend the rest of it was absorbed: the DX helpers in This was one of the earliest external contributions to the TypeScript port and it sat open while that port changed underneath it, which is on the repository rather than on you. Sorry it went that way. |
Summary
Makes
laya-tsfaster for bulk workloads and easier to integrate, without changing any existing API semantics. Six themed commits, each building and passing tests in isolation:fix(ts)— loop-based tensor plumbing, cures spread overflowperf(ts)— BPE, question-prefix, and hooks fast-path cachesfeat(ts)—Agent.predictMany/Router.predictManybatch inferencefeat(ts)—qbuilders, result helpers, shortlist auto-embedfeat(ts)— retrying loads with abort signal and progressdocs(ts)— working examples and batch READMEMotivation
Math.max(...lengths)throwsRangeErrorpast ~100k args, so large batches crashed instead of running.[B,L,H]hidden tensor up to 3x per call.predictShortlistforced callers to hand-rollembed_fndespite the agent carrying an encoder../model-ml, which exists in no checkout, so every example failed out of the box.What changed
Perf / fixes (
common.ts,providers.ts,shortlist.ts,tokenizer.ts,hooks.ts)maxOf(values, fallback)loop helper;collateItems,feed,feedHead, and shortlist embed widths use it.toNestedcopies once with precomputed steps;i64,hidden_states, andmarker_maskflatten straight into typed arrays (noflat(Infinity).mapintermediates). Values and dims verified identical bytests/flat-parity.test.ts.composeHooksshared-empty fast path. All transparent to callers.Batch API (
agent.ts,router.ts)predictMany/systemManyon both classes. Agent runs one encoder+head pass over all SxQ rows;_systemOneCoredelegates to the same core, so single and batch paths can't diverge. Hook semantics unchanged (rewrite/skip/usage aggregation preserved).onRoute), groups by model, loads each checkpoint once, preserves input order, attaches per-staterouting. Agents withoutpredictManyfall back to loopedsystemOne.DX (
dx.ts,index.ts,shortlist.ts)q.choice/q.score/q.noulfactories (output passescheckQuestion),best(),topK(),isConfident(); all exported from the package root alongsidemaxOf.predictShortlist(agent, state, questions)now defaults toembedFnFromAgent(agent)when shortlisting is actually needed; explicitembedFncallers unaffected.Load robustness (
providers.ts,agent.ts)fetchWithRetry(2 retries on network errors and 429/5xx, 404s fail fast, aborts never retried);signal+onProgress(done, total, file)throughloadNodeBundle,loadWebBundle, andAgent.load; versioned CacheStorage key (laya-ts-v1).Examples / docs
examples/try-batch.mjs(one-pass batch + routed batch with per-state choice/confidence output);try-ml.mjsandsnake.mjsresolve the shipped weights via--modelplus first-existing-dir fallback; README batch snippet.Verification
maxlen,flat-parity,prefix-cache,batch,dx,load-robustness); every commit was checked out in an isolated worktree with a cleantscbuild and a green suite.batched: billing/billing/other,routing: english/multilingual/multilingual;try-ml.mjsreports Hindi refund -> billing 0.99.Notes / deliberate non-changes
QuestionPrefix.nOptions(exported type) and thePredictHookArg/HookArgsplit (callables valid asonPredictStartbut correctly rejected ashooks); skipped a lang single-pass and cache TTL as not worth it at these sizes.